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SUMMARY 

The QIBB quality initiative was specifically developed for the vocational 
training system in Austria. The quality system is distinctive in that it 
includes both objectives and survey instruments designed to address 
the specific concerns of vocational training. There are general quality 
concepts and instruments that are common to all schools and others 
that are differentiated according to the type of school. The various school 
types within the vocational training system use QIBB first of all to conduct 
system-related surveys, for example to test common quality objectives 
in connection with national priorities. QIBB also enables schools to 
conduct internal evaluations. Survey instruments and an automatic 
evaluation are available from an online database. The test power of the 
survey instruments made available in QIBB are also verified in empirical 
analyses. Initial studies indicate that the test power of almost all survey 
instruments examined is satisfactory. 
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Definition of school quality 

If a group of people is asked what they understand by ‘school quality’, 
there will probably be as many answers as there are people. In fact, 
the question cannot be answered with reference to a single indicator 
and, depending on the approach, different indicators will be arrived 
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at, focusing, for instance, on outcomes such as performance or 
competences or on targets such as human resources and budget 
(see Bottcher, 2002; Fend, 2001 ; Kempfert and Rolff, 2005; National 
Education Association, 2008; Schratz, Iby Radnitzky, 2000). The 
European Commission defines the core aspects of school quality 
as follows: knowledge, success and transition (to further education), 
monitoring education and resources and structures (internal school 
resources and structures; SEQuALS, 2008). 

Classification of indicators of school quality based on a multi- 
level model 

Given the variety of indicators of school quality, a classification 
system needs to be used to categorise quality indicators. The multi- 
level model is such a system in that it sorts indicators according to 
different levels within the educational system (Fend, 2000, 2001): 

• level of the school system: this level comprises indicators 
such as equal opportunities and supply density as well as 
humanitarianism (regulation of freedoms, participation and 
obligations) (Fend, 2001); 

• school level: this is where indicators such as social climate 
and working atmosphere in school, satisfaction, student and 
parent participation or the quality of cooperation with the school 
environment (Fend, 2001; Holtappels, 2003) are to be found. 
These are indicators that concern the school system as a 
global system within which different people move and which is 
established in a social environment; 

• class level: at this level indicators such as performance, 
learning and social behaviour, social relationships and 
interaction between students and teachers as well as parents 
and teachers are located, i.e. indicators that view the class as a 
learning community embedded in a social environment (Fend, 
2001; Holtappels, 2003); 

• personal level: this level comprises indicators relating to 
people. For teachers, this includes competences and attitudes; 
for students, it comprises attitude to study and work and 
competences (Fend, 2001 ; Holtappels, 2003). 

Classification of indicators for school quality based on a process 
model 

If educational and training processes are viewed as a chronological 
sequence in which it should be possible ‘to turn schools, for all 
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students if possible, into productive spaces for learning and their 
long-term development’ (Fend, 2000, p. 56), then indicators for 
school quality can be classified according to whether they relate 
to the framework conditions for the promotion of development 
processes, the actual teaching, learning and development 
processes or to training and education outcomes. 

If school quality is related primarily to the outcomes of 
education and training processes, first of all quality indicators can 
be identified on the part of students or graduates. They include 
not only learning achievements, technical and interdisciplinary 
skills acquired, personal characteristics and attitudes, but also 
satisfaction with schooling and the atmosphere in the class or 
school. Secondly, outcomes can be identified on the part of the 
parents or the school’s cooperation partners, such as parent 
satisfaction with training or the assessment of training outcomes 
by cooperation partners such as training firms. 

At the process level, various indicators can be discerned relating 
to the design of organisational, teaching, learning and tuition 
processes. This category covers teacher-student interaction, 
support measures and, at school level, personnel development 
and employee promotion. 

By contrast, conditional structures relate to the structural and 
procedural constraints under which a school - but also the various 
institutions within an educational system in general - organises 
its work. The conditional factors at school level include, for 
example, not only aspects such as formal performance standards, 
organisational framework and type of school but also external 
conditions (for example, the job market; Holtappels, 2003). 


The vocational school system in Austria 

In Austria, compulsory education begins at the age of six and 
lasts nine years. Secondary school education up to the school- 
leaving examination is called Sekundarstufe II in German-speaking 
countries. This educational segment is highly subdivided in Austria: 
the academic secondary school ( Gymnasium in Sekundarstufe II) is 
distinct from the vocational school in terms of its curricular content 
and objectives. The vocational school system is divided into part- 
time vocational schools ( Berufsschulen ) for training as part of an 
apprenticeship, and vocational middle and upper schools. The training 
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corresponds with levels 3b to 5b of the international classification 
ISCED 97 (UNESCO, 2008), (http://portal.unesco.org). 

The Austrian vocational middle and upper schools cover a 
range of subjects. The following school types are classified by 
specialisation: 

• technical, trades and art colleges; 

• business colleges; 

• colleges for occupations in the social and services sector (spe- 
cialising in fashion, tourism, social services and professions); 

• colleges for agriculture and forestry; 

• kindergarten training colleges and colleges for social pedagogy. 
The vocational schools also participate in the quality initiative. 

Besides vocational training, they impart theoretical and general 
knowledge and complement company-based training (bm:ukk, 
2008a). 

In the school year 2003/04, 25.6 % of Austrian students in the 
tenth grade - i.e. directly after finishing compulsory education - 
attended vocational upper schools and 1 3.2 % attended vocational 
middle schools. 18.8 % attended academic secondary schools 
( Gymnasium ) and 36.9 % part-time vocational schools. 


QIBB quality initiative of the vocational school 
system in Austria 

Launching the ‘VET quality initiative - QIBB’ (QIBB, 2008a) in 2004 was 
a significant step towards quality development and quality assurance 
in the Austrian school system. This joint initiative of all Austrian 
vocational schools marked the development and implementation 
of a quality system that comprises schools, the school supervisory 
board and the Federal Ministry for Education, Arts and Culture as the 
highest education authority, and perceives, defines and procedurally 
describes quality as a complete cycle - from guiding principles through 
to evaluation and modification measures. 

The quality system of the part-time vocational schools 

Quality manuals set out the organisation of the content as well as 
the features of the quality system (Paechter and Mayringer, 2006; 
QIBB, 2008a, 2008b). Key elements of these manuals are the 
mission statement, the quality matrix and guidelines on the content 
of the annual quality reports. 
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The mission statements express the common self-conception 
of the training institutions involved. This self-conception is marked, 
irrespective of the individual school type, by a close alignment on 
the demands of individual professions. In the quality matrix, the 
quality fields mentioned in the mission statements are broken down 
into sub-objectives. Each sub-objective is assigned (pedagogical, 
didactic and organisational) measures, outputs and indicators for 
the purposes of evaluation. A further feature of the quality system 
is that the quality work is conceived of as a continuous, objective- 
oriented and feedback-based improvement process (Federal Ministry 
for Education, Arts and Culture, 2008b). In line with this approach, 
the guidelines, quality objectives and sub-objectives, measures, 
desired outputs, indicators and evaluation methods and instruments 
form the framework of the quality work. 

The quantity and wording of the quality fields, measures, outputs 
and indicators can vary according to school type. However, the 
quality matrices of all school types contain quality fields from four 
basic areas: 

• teaching and learning: this area is divided into different quality 
objectives. They concern issues such as the quality of vocational 
training, general and personal development through school and 
support for students in a stimulating study and work environment; 

• administration and quality management: the objectives in this area 
address employee advancement, quality development in relation 
to the performance of educational institutions and participation 
in quality development measures; 

• economy and society: objectives in this area concern the 
innovative potential of training programmes and ensuring practical 
relevance through theoretical and practical training, the practical 
experience of teachers and cooperation with industry and other 
cooperation partners; 

• internationality: this area concerns the international dimension 
of the training programmes and the opportunities to encourage 
mobility and intercultural understanding among the students. 
With the exception of ‘internationality’, all the objectives formulated in 

the quality areas fall within the four categories defined by the European 
Commission as the core areas of school quality. The quality system 
employs a multi-level approach in so far as it specifies, for each 
objective (with output and indicators), the level within the school system 
where appropriate measures for quality development and assurance 
can be implemented (school authorities or the school itself). 
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Since the quality objectives are derived from and further differentiate 
the mission statements, there is always a broad spectrum of quality areas 
and a relatively large number of sub-objectives. For practical reasons, 
it is not feasible for all sub-objectives to be evaluated every year. Each 
year, therefore, a steering committee of school authority representatives 
and head teachers sets evaluation priorities as binding quality areas. 
Moreover, each school can work on other quality areas. 

Evaluation 

In the feedback-based quality system, objective-oriented, systematic, 
regular evaluation at all system levels is a prerequisite for quality 
development and assurance. This is based on the following guiding 
principles (see Holtappels, 2003): 

• the evaluation meets the criteria of utility, feasibility, objectivity 
and correctness; 

• it fulfils ethical principles; 

• the evaluation is performed on both the processes and the 
outcomes of pedagogical and administrative action; 

• feedback is collected from all groups capable of directly or 
indirectly evaluating the quality of the school; 

• the survey instruments made available have the requisite test 
power. 

The evaluation of part-time vocational schools in Austria exhibits 
characteristics of both external and internal evaluation. It is external 
in so far as evaluation instruments such as questionnaires are 
supplied to schools and then evaluated externally in an online 
environment. A further feature of external evaluation is that an 
evaluation is supposed to be made of selected quality objectives 
for all schools. An internal evaluation characteristic is that schools 
can define further evaluation areas for themselves, but must use 
the specified survey instruments. Another aspect that corresponds 
to internal evaluation is the fact that schools themselves prepare 
a quality report, that is to say they can interpret and comment on 
evaluation results (see Kempfert and Rolff, 2005). Also as regards 
the question of self-evaluation versus external evaluation, the 
present system uses a mix of methods. Teachers evaluating their 
own lessons is clearly self-evaluation. If, however, outsiders, such as 
representatives of industry, provide assessments, that corresponds 
to external evaluation (see Holtappels, 2003). 

Although schools bear the lion’s share of evaluation and are 
viewed as the key players in quality development, thinking about an 
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overall system is clearly to the fore. For evaluation, this means that 
it must be possible to compare and aggregate the results in parts. 
That is why schools are provided with a set of specially developed 
evaluation instruments and with electronic support for conducting 
and evaluating the surveys. 

Survey instruments 

To assist schools with their evaluations, survey instruments were 
developed that are tailored to the requirements of part-time vocational 
schools. They include the following types of instruments: 

• questionnaires for students, graduates, teachers and parents; 

• guidelines for discussions, e.g. with company representatives; 

• survey forms for the school, e.g. for statistical surveys of the 

facilities and human resources. 

Like a process model, the instruments target various quality 
indicators: processes (in the school, in the classroom) and behaviours 
(school management, teachers, administrative staff and students), 
outcomes/attitudes (students, teachers) and structural factors 
(facilities and human resources). 

In terms of their specificity, three types of survey instrument can 
be identified: first, there are instruments that can be employed for all 
school types; second, there are instruments that are suitable for all 
school types in their basic form, but which contain some questions 
(e.g. regarding subjects) that are specifically formulated for each 
school type; finally, there are instruments that cover the specific 
structures, processes or results of a specific school type. 

The survey instruments of the quality system are designed to 
ensure that statements or assessments in evaluations made by 
others never refer to an individual. Students do not assess an 
individual teacher but rather all their teachers as a whole (example 
of a questionnaire item: ‘my teachers advise me how to improve 
my performance’ - assess on a scale of (1) ‘strongly agree’ to (6) 
‘strongly disagree’). Thus, already at the survey stage, the data 
are not obtained to assess individuals. If possible, each aspect is 
assessed by a number of groups, e.g. by teachers and students. 
The survey instruments, which always cover the behaviour of a 
group of persons, are used to evaluate quality objectives within the 
context of a system evaluation. In addition, individual schools can 
also use the instruments for an internal evaluation. 

QIBB also contains questionnaires which individuals can use 
to obtain feedback on their own professional behaviour. Such 
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Table 1 . Overview of topics covered by the survey instruments, 
ordered by processes and levels 


1 Assessor/Instruments 

Structural factors 

Process factors 

Result factors 

Students 

(questionnaires) 

Environment/school: 
school-entry phase 
(lower classes) 
School: facilities 

Class: course 
design (variety of 
methods, gender 
fairness, performance 
appraisal, practical 
relevance, etc.) 

Persons: attitudes, 
competences 
Class: atmosphere 
School: atmosphere 

Graduates 

(questionnaires) 

— 

— 

Persons: attitudes, 
competences 

Teachers 

(questionnaires) 

School: facilities, 
resources, 
management of 
organisational and 
administrative 
processes 

Persons: Course 
design 

School: personnel 
development, 
collaboration with 
colleagues 

Persons: 

satisfaction 

Parents 

(questionnaires) 

Environment/school: 
school-entry phase 
(lower classes) 

Class: interaction 

School: atmosphere 

School 

(survey forms) 

School: facilities, 
resources, curricula, 
training courses, 
management of 
organisational and 
administrative 
processes 

School: personnel 
development, 
resources, 
cooperation with 
external partners 

Persons: 
certificates, final 
exams of students 

External partners 
(businesses, etc.) 
(discussion guidelines) 

School: curricula 


Persons: 
competences of 
graduates 


questionnaires are available for each level within the school system 
(teachers, head teachers, school inspectors, etc.)- In this way, 
teachers can have their teaching assessed by the students. 

To conduct and evaluate the surveys, schools are provided with 
an efficient technical infrastructure financed by the school authorities. 
All survey instruments are available as Web forms that can be 
processed and evaluated online. 

Verification of the test power of the survey instruments using an 
illustrative sample 

During the development of the QIBB survey instruments it was not 
possible to draw on existing instruments from other surveys or school 
experiments. There were a number of reasons for this: for example the 
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quality objectives are specifically tailored to part-time vocational schools. 
By contrast, initiatives in other countries often view school quality from 
a global perspective that considers only quality elements common to 
all school types and school levels (for Germany see: Basel and RQtzel, 
2005; Netzwerke innovativer Schuien und Schulsysteme, 2008 ; Zoller, 
2005; for Switzerland: QuiSS, 2008 ; for the United Kingdom: HMIe, 
2008). Such a system would not do justice to the distinctive features of 
the vocational training system. Furthermore, since there were no models 
from the vocational training system, not only the quality system itself 
but also the survey instruments had to be developed from scratch. 

This means that the test power of the new instruments had to 
be verified in an empirical study. Thus when the evaluation was 
first conducted in the 2005/06 school year, the test power of 12 
questionnaires on different fields (e.g. transparency of performance 
appraisal, gender-neutral teaching, administrative processes, and 
personnel development) was verified. The following questions were 
germane to the appraisal of the test power of the questionnaires: 

• do the questionnaires or the questions in the individual 
questionnaires differentiate between varying types of quality? 

• is test power of the questionnaires adequate? This question 
concerns, for instance, whether the items within a questionnaire 
show inherently valid structures in an empirical test; 

• how strongly do evaluations of identical aspects vary from group 
to group (e.g. teachers and students)? 

In the following, the verification of the test power of the instrument 
‘transparency of performance appraisal’ will be presented by way of 
example. For this quality objective, one questionnaire was designed 
for students and one for teachers, although the teachers’ questionnaire 
was not used in every type of school. The analysis draws on data for 
three school types from the 2005/06 school year, the first year of the 
evaluation. These types comprise schools and colleges for occupations 
in the social and service sector, colleges of business administration 
and business schools, and colleges for engineering, arts and crafts. 
The other school types began the evaluation a year later. 

Review of the test power of the students’ questionnaire on 
‘Transparency of performance appraisal’ 

In the 2005/06 school year, 70 355 students participated in the online 
survey, of whom 36 21 7 attended secondary schools and colleges 
for occupations in the social and services sector [humanberufliche 
Schuien] (1 737 classes, 51.48 % of participants) and 34 138 
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attended colleges for engineering, arts and crafts (1 800 classes, 
48.52 % of the participants). 

Table 2 lists mean values, standard deviation and the number of 
cases for each item of the questionnaire. 

Table 2 shows standard deviations of the items with values between 
1 .18 and 1 .45. The rather high standard deviations indicate that the 
students’ assessments span the entire scale. This is desirable because 
the items in the later evaluation are intended to differentiate between 
various characteristics (e.g. in teaching behaviour) and facilitate 
comparisons between various groups and classes within a school. 

The first step in examining the test power of the questionnaire 

Table 2. Mean values, standard deviation and number of cases 
for each item of the questionnaire ‘Transparency of 
performance appraisal (students)’ 


No item M SD N N/A 


1 

At the start of the school year, the teachers explain to us the 
knowledge and skills we are supposed to acquire by the end 
of the year. 

2.92 

1.36 

70 343 

12 

2 

The teachers discuss with us why we require those particular 
teaching materials. 

3.73 

1.43 

70 335 

20 

3 

The teachers make us aware of interconnections with other 
areas of life and learning. 

3.04 

1.29 

70 334 

21 

4 

My class teachers tell us how marks are determined 
(e.g. weighting of questions) in individual appraisals (e.g. 
schoolwork, test, oral examination). 

2.43 

1.34 

70 331 

24 

5 

My class teachers tell us at the start of the school year how 
the overall mark is determined for each subject (e.g. weighting 
of individual performance, importance of class participation). 

2.30 

1.29 

70 331 

24 

6 

My class teachers provide timely information on examination 
dates and submission deadlines (e.g. for schoolwork, tests, 
project work, homework). 

2.05 

1.18 

70 329 

26 

7 

My teachers update me on my current performance level if 1 
ask them to. 

2.40 

1.36 

70 325 

30 

8 

The marks 1 am given reflect my actual performance. 

2.64 

1.28 

70 315 

40 

9 

My teachers advise me how 1 can improve my performance. 

3.07 

1.45 

70 292 

63 

10 

The teachers separate advice and feedback from the 
performance review. 

2.98 

1.30 

70 082 

273 


M=mean value; SD=standard deviation; N=sample size; N/A=number of missing values; all items were 
answered on a scale of 1 (strongly agree) to 6 (strongly disagree). 
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was to investigate whether the individual items were answered in a 
differentiated manner. To do this, the so-called ‘item popularity’ was 
ascertained across the sample of all students (Buhner, 2004). This 
measurement shows if a question is answered with a desired variance 
or if all answers cluster around a similar point on the scale so that 
no differentiation can be made between different qualities. An item 
popularity of between 0.2 and 0.8 is desirable. In fact, examination 
of the ten items revealed satisfactory item popularity for all items. 

The second step was to analyse relationships between items. 
To this end, a factor analysis was conducted on all items. During 
the development of tests and questionnaires, factor analyses 
are computed to check relationships between items and the 
dimensionality of a survey instrument. Thus, an extensive number 
of items can be broken down into homogeneous subcategories 
and analysed to determine if a number of items include a common 
latent variable (Buhner, 2004). Since the questionnaires described 
here were not developed on the basis of an existing and verified 
theoretical model, an exploratory factor analysis was computed 
(Buhner, 2004). It seeks to aggregate the items into groups on the 
basis of the correlations empirically found to exist between them. 
Table 3 shows the statistical values of the factor analysis. 

Two factors were extracted in the factor analysis: items 1 , 2 and 
3 point to high loads on a common factor (between 0.67 and 0.80; 
see evaluation of factor loadings in Bortz, 1999). All three items 
concern whether teachers explain the usefulness of teaching material 
in real life and the marking system (which is why the factor is called 
‘teaching material’). All other items (4 to 9) load heavily on a second 
factor with weights ranging from 0.56 and 0.69. This group of items 
focuses directly on performance appraisal (‘performance appraisal’ 
factor). The joint explained variation of the two factors is 49.8 %. The 
factor analysis thus shows a very clear structure with two factors 
that can also be explained by the content of the items. The data 
of all school types were combined for the factor analysis. A more 
comprehensive review was carried out to investigate whether the 
same factor structures are found in the different school types. It was 
possible to confirm that they are (Paechter and Lunger, 2007a). 

Verification of the test power of the questionnaire for teachers on 
‘Transparency of performance appraisal’ 

As regards teaching staff, a total of 3 618 teachers took part in the 
survey; 3 034 from the engineering and industrial colleges (83.9 %), 319 
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Table 3. Factor loadings of the questionnaire Transparency of 
the performance appraisal (students)’ 


No 

Item 

Load on factor 
1 2 

1 

At the start of the school year, the teachers explain to us the knowledge 
and skills we are supposed to acquire by the end of the year. 

0.16 

0.75 

2 

The teachers discuss with us why we require those particular teaching 
materials. 

0.16 

0.80 

3 

The teachers make us aware of interconnections with other areas of life 
and learning. 

0.23 

0.67 

4 

My class teachers tell us how marks are determined (e.g. weighting of 
questions) in individual appraisals (e.g. schoolwork, test, oral examination). 

0.60 

0.30 

5 

My class teachers tell us at the start of the school year how the overall 
mark is determined for each subject (e.g. weighting of individual 
performance, importance of class participation). 

0.56 

0.32 

6 

My class teachers provide timely information on examination dates and 
submission deadlines (e.g. for schoolwork, tests, project work, homework). 

0.69 

0.05 

7 

My teachers update me on my current performance level if 1 ask them to. 

0.68 

0.08 

8 

The marks 1 am given reflect my actual performance. 

0.68 

0.16 

9 

My teachers advise me how 1 can improve my performance. 

0.58 

0.38 

10 

The teachers separate advice and feedback from the performance review. 

0.57 

0.36 


from secondary schools and colleges for occupations in the social and 
services sector (8.8 %) and 265 from business schools (7.3 %). 

Apart from numbers 8 and 10, the items in the questionnaire 
for students and teachers have parallel formulations (from the 
perspective of the student or teacher respectively). 

Table 3 shows that, compared to the sample of students, the 
teachers’ self-assessments (as expected) are far more positive (mean 
values between 1 .24 und 2.37). The standard deviation of the items 
is also considerably lower. Moreover, the item popularities were lower 
than those of the correspondingly worded items for students. 

Tests were also conducted to see how the various questionnaire 
items relate to one another and an exploratory factor analysis on 
all items was computed. 

The factor analysis shows two groups of items that match the structure 
of the questionnaire for students. As in the students’ questionnaire, 
items 1, 2 and 3 are grouped into one factor (‘teaching materials’). 
All items have a high factor loading (between 0.56 and 0.80). Items 
4, 5, 6, 7, 8, 9 can be grouped together into another factor (‘appraisal 
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transparency’). They have a medium to high factor loading of 0.47 to 
0.68. Item 1 0 comprises a factor on it own. No - loads on both factors 
- does not have a simple structure - cannot be interpreted. In terms 
of content, it cannot be assigned to the other items. However, it was 
left in the questionnaire for future use because it reflects an important 
aspect of the professional behaviour of a teacher. 

Table 4. Mean values, standard deviation and case numbers 
for each item of the questionnaire ‘Transparency of 
the performance appraisal (teachers)’ 


No 

Item 

M 

SD 

N 

N/A 

1 

At the start of the school year, 1 explain to my students which 
knowledge and skills they are supposed to acquire by the end 
of the school year. 

1.70 

0.92 

3 615 

3 

2 

1 discuss with my students why they require those particular 
teaching materials. 

1.67 

0.85 

3 615 

3 

3 

1 make my students aware of interconnections with other 
areas of life and learning. 

1.59 

0.77 

3 615 

3 

4 

1 tell my students how the marks are determined (e.g. 
weighting of questions) in individual appraisals, (e.g. 
schoolwork, test, oral examination). 

1.42 

0.75 

3615 

3 

5 

1 tell my students at the start of the school year how the 
overall mark is determined for each subject (e.g. weighting of 
individual performance, importance of class participation). 

1.39 

0.73 

3 615 

3 

6 

1 provide my students with timely information on examination 
dates and submission deadlines (e.g. for schoolwork, tests, 
project work, homework). 

1.24 

0.63 

3 615 

3 

7 

1 update my students on their current performance level if they 
ask me to. 

1.30 

0.63 

3 614 

4 

8 

1 use the information and early warning system consistently 
and in good time. 

1.64 

0.96 

3612 

6 

9 

1 advise my students how they can improve their performance. 

1.63 

0.84 

3 603 

15 

10 

1 stay abreast of the legal provisions governing performance 
appraisal. 

2.37 

1.31 

3 579 

39 


M=mean value; SD=standard deviation; N=sample size; N/A=number of missing values; all items were 
answered on a scale of 1 (strongly agree) to 6 (strongly disagree). 


The factor structure for the teachers’ questionnaire is on the 
whole very clear and intrinsically consistent. Nevertheless, this 
survey instrument should be viewed with caution. The mean values 
and the low standard deviations of the items show that the teachers 
exercise little self-criticism in assessing their own behaviour, and their 
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judgement is on the whole uniform. Used alone, this questionnaire 
does not lend itself to assessing the quality of teaching. It should 
instead be used in conjunction with the students’ questionnaire as 
a complementary instrument. 

Comparison of the item mean values for students and teachers 
A comparison of the item mean values in the questionnaire for 
students and teachers can shed light on the extent to which the 
assessments are similar. It is to be expected that the students’ 
assessments will be ‘stricter’ since they represent an assessment of 
other people, whereas the teachers are judging their own behaviour. 
It would be preferable, for each item, for the assessments to run 
in parallel as far as possible. Figure 1 shows, for the items with a 
parallel formulation, the mean values of the assessments of both 
students and teachers. 

As expected, the teachers’ assessments are lower than those of 
the students. As an assessment of their own actions, they are more 
positive than the external assessment of the students. Apart from 


Table 5. Factor loadings of the questionnaire ‘Transparency of 
the performance appraisal (teachers)’ 



Load on factor 

No Item 

1 2 


1 

At the start of the school year, 1 explain to my students which knowledge 
and skills they are supposed to acquire by the end of the school year. 

0.19 

0.70 

2 

1 discuss with my students why they require those particular teaching 
materials. 

0.05 

0.83 

3 

1 make my students aware of interconnections with other areas of life and 
learning. 

0.12 

0.65 

4 

1 tell my students how the marks are determined (e.g. weighting of 
questions) in individual appraisals, (e.g. schoolwork, test, oral examination). 

0.68 

0.13 

5 

1 tell my students at the start of the school year how the overall mark is 
determined for each subject (e.g. weighting of individual performance, 
importance of class participation). 

0.61 

0.22 

6 

1 provide my students with timely information on examination dates and 
submission deadlines (e.g. for schoolwork, tests, project work, homework). 

0.68 

0.00 

7 

1 update my students on their current performance level if they ask me to. 

0.53 

0.17 

8 

1 use the information and early warning system consistently and in good time. 

0.64 

0.12 

9 

1 advise my students how they can improve their performance. 

0.47 

0.43 

10 

1 stay abreast of the legal provisions governing performance appraisal. 

0.40 

0.40 
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item 2, however, all items for both the teachers and the students 
are positive (scale range of 1 to mid-scale 3.5). The assessments 
of students and teachers largely run in parallel. 

Conclusion of the verification of the test power of the survey 
instruments 

The statistical analyses outlined above were carried out for 1 2 survey 
instruments in order to verify the test power of the questionnaires. 
On the whole, the analyses show that the test power of almost all 
instruments is satisfactory. In the case of only one instrument, it was 
clear that it could be shortened by eliminating a number of similarly 
formulated items (Paechter and Lunger, 2007a, 2007b). 

Figure 1 . Assessments by students and teachers (mean values 
by item) 
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